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ABSTRACT 
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hardly addressed by these indicators- Instead, more tangible measures 
of input and output predominate. The quality of learning arising from 
the experience of higher education is rarely considered, although any 
valid measure of a uni^rsity ■ s performance must take account of the 
intellectual development of its students. A combination of 
qualitative and quantitative evaluations is necessary to the process 
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Abstract 



Governmental pressures for universities to demonstrate greater efficiency while 
maintaining the quality of their activities have resulted in a range of 
performance indicators being adopted. However, questions of quality in 
teaching, learning and research are hardly addressed by these indicators: Instead, 
more tangible measures of input and output predominate. The quality of learning 
arising from the experience of higher education is rarely considered, although 
any valid measure of a university's performance must take account of the 
intellectual development of its students. A combination of qualitative and 
quantitative evaluations is necessary to the process of monitoring standards and 
performance in higher education. 
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Learning: The Forgotten Dimension? 



The Background. 

Throughout the 1980*s there has been mounting pressure from governments in 
Europe and elsewhere for institutions of higher education to demonstrate greater 
efficiency in the utilisation of resources, while at the same time maintaining the 
quality of teaching and research. The British Government's policies for 
increased public accountability in the university sector have been mediated 
through bodies such as the Committee of Vice-Chancellors and Principals and 
the Universities Grants Committee (e.g. Committee of Vice-Chancellors and 
Principals 1987), resulting in the development and publication of a range of 
performance indicators for use in universities. These indicators are primarily 
intended to allow comparisons to be made between institutions, although they 
may be supplemented or modified to facilitate the comparison of departments 
within an institution. Recommendations concerning the maintenance and 
monitoring of standards in universities were published in a separate document 
(Committee of Vice-Chancellors and Principals 1986). It will be argued that, in 
both of these management led movements, a key issue - the quality of students' 
learning arising from the experience of higher education - has been overlooked if 
not forgotten altogether. 

The Quest for Greater Institutional Efficiency. 

The performance indicators listed by the C.V.C.P. are wholly administrative, as 
they concentrate on factors that relate to efficiency in the utilisation of resources 
rather than to educational quality and standards. Academic questions of quality 
in university teaching, learning and research have hardly been addressed: 
Instead, the more tangible and largely managerial measures of input and output 
have been adopted. These may be easier to quantify and to collect than those of a 
more qualitative (and hence more elusive) nature, but entail the imposition of 
values and objectives from outside the academic community. The emphasis 
upon managerial indicators of performance may provide greater control, but 
may also prove to be detrimental to the achievement of educational quality and 
effectiveness (Pollitt 1987). If institutions of higher education are increasingly to 
be regarded as 'production enterprises', we must be clear what the 'products' are, 
or risk producing - with great efficiency - a totally inappropriate output. 

Some universities have attempted to develop indicators of pniormance that 
address the issue of educational effectiveness (see, for example, Rutherford 1987 
and West 1988). However, these are generally concerned with the appraisal of 
formal curricula or of observable teaching activities and artefacts rather than the 
process or content of student learning. Output measures such as graduation 
rates, numbers of 'good' degrees attained and graduate employment patterns are 
fraught with difficulties when used as the basis for appraisal or comparison: 
These difficulties include 



• considerations of the 'value added' by the experience of higher education, 
depending upon entrants' differing levels of ability and the contribution of a 
particular institution to the final outcome; 

• the subjectivity of notions of what actually constitutes a 'good' degree and the 
normative nature of much student assessment; 

• the risk of being over-concerned with the numbers of graduates to the neglect 
of what they are able to do or achieve in terms of intellectual and other skills; 

• the limited validity of monitoring graduate employment in terms of first 
occupation, as many graduates change career direction after their first 
appointment. 

Furthermore, all fuch measures face problems in terms of mature-aged students 
and/or those lacking the normal entrance requirements for university study; m 
other words, the very people that are likely to represent a Rowing proportion of 
U.K. higher education students in the 1990*s and beyond (Department of 
Education and Science 1987). , , 

Later in this paper it will be argued that the quality of student learning does not 
necessarily correlate with the llvel of degree attained, nor with graduation per se. 
Any valid measure of a university's performance must take account of tne 
intellectual development of students in higher education. 

M aintaining Standards in Hi g h ? r Education. 

The British Government stated in its most recent White Paper on Higher 
Education that "Quality in higher education depends primarily upon the 
commitment of the academic community to the maintenance and improvement 
of standards" and has indicated that it will "seek to ensure that systems are m 
place to promote and give effect to that commitment and to monitor the results 
toeparttnent of Educarton and Science 1987, p. 16) The guidelines on academic 
standards published by the CV.C.P. (Committee of Vice-Chancellors and 
Principals 1986) represent a number of procedures that the academic community 
can utilise to monitor and maintain the quality of their activities. 

Although these guidelines (relating to the planning of new courses and to the 
monitoring of existing ones) should be welcomed, procedures m themselves can 
neither engender nor ensure the commitment of the academic community to 
improving standards. In particular, the procedures remain vague about the 
purposes of higher education and how best they can be achieved. The academic 
conununity hat tended to take such issues for granted, but it is increasingly dear 
that t rposes (and the means of achieving them) must be explicitly stated and 
discussedat a time when administrative/managerial concerns appear to prevail. 
Through the explicit consideration of such fundamental issues, universities can 
demonstrate their capacity for promoting and maintaining academic quality 
(Barnett 1986). 

Much rhetoric surrounds the outcomes of undergraduate education: As well as 
acquiring new knowledge and skills, we want our students to demonstrate 
'critical thinking', 'high level problem solving', 'independent intellectual 
inquiry' or to understand and conceptualise phenomena in the manner of a 
particular discipline or profession. The long term effects of higher education 
have more to do with intellectual development and the processes of learning 



than with the particular content (see, for example, Powell 1985 and Taylor, 
Morgan and Gibbs 1983). However, recent research on students' experiences of 
learning suggests that there is often a lack of congruence between stated course 
aims and the actual curriculum presented, particularly the demands of the 
assessment procedures (see, for example, Marton, Hounseil and Entwistle 1984). 
In examinations, low level cognitive demands (the recall of factual information 
and the rehearsal of well-established procedures) tend to predominate and shape 
the ways in which students undertake learning tasks. The quantitative criteria 
applied in much student assessment (concentrating on haw much students know 
rattier than how well they understand) make it possible for students to gain 
enough marks to pass without actually developing real understandings of basic 
ideas and principles or changing the way they conceptualise important 
phenomena. 

Qualitative research on student learning (developed from the work of Marton 
and his associates) has not only provided a framework for examining and 
understanding the learning processes in higher education and their outcomes, it 
has also shown how the quality of undergraduate courses can be improved. "We 
must learn to celebrate process as well as product in student learning in order to 
enhance its standard: to see effectiveness in terms of how students manipulate 
tasks and conceptualize content" (Ramsden 1986). Monitoring in an academic 
community should ensure that learning tasks are appropriate for the purposes of 
university education and that student support and assessment practices 
encourage the promotion of students' intellectual development. 

Combining Qualitative a nd Quantitative Evaluations. 

In trying to assess the quality and performance of institutions of higher 
education, it is suggested that quantitative measures (that tend to be favoured by 
administrators and policy makers) can provide only a part of the picture, while 
qualitative evaluations (favoured by university teachers and researchers) may 
give greater validity to the exercise but are more difficult to use on a comparative 
basis A combination of qualitative and quantitative evaluations is necessary to 
the process of monitoring and maintaining standards and performance in higher 
education. The following account of routine monitoring procedures and other 
evaluative studies undertaken at the Open University provides an illustration of 
•quality assessment and control* within a collaborative and self-critical 
community of academic and support staff. 

'Quality Con trol' at the Ooen University - A Case Study. 

The Open University (O.U.), now the largest university in the U.K., was 
established twenty years ago to provide part-time higher education for adults 
studying in their own homes. The university has no entrance requirements in 
terms of certification from previous education and many students are in full- 
time employment. It has a modular degree structure, offering students an 
almost free choice from over one hundred full- and half-credit courses that can 
be accumulated to make up an individual's degree profile; six credits for the B.A. 
degree, eight credits for a B.A. Honours degree. 

Teaching at a distance involves academic staff and students being separated, both 
geographically and temporally, such that instruction is largely mediated through 
written or recorded materials. All courses comprise teaching texts and audio- 
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visual materials (supported by set readings, practical activities, assignments, etc.) 
that have been specially prepared by a team of academic and other staff working 
in collaboration. The 'published' materials developed in this way normally 
remain in substantially the same form for several cohorts of learners, although 
some modifications and amendments can be made when necessary. It is 
tempting to view the teaching materials as the 'products' of the university - they 
are visible and can be (and are) inspected by the public at large. However, it is 
what students learn from interacting with our materials and with tutors and 
fellow students that constitutes the 'product' of our institution. The separation 
of teachers from learners also entails the absence of direct verbal and non-verbal 
feedback from students that is so important in all teaching. Systematic 
procedures had to be devised to provide our students with opportunities to 
comment on the quality of the courses they were studying and on the system of 
distance education with which they were engaged. In terms of monitoring and 
maintaining academic standards, it was also necessary to establish procedures to 
obtain the evaluative information we required from others in the higher 
education community. The procedures utilised at the Open University combine 
qualitative and quantitative methodologies to provide a complex picture of the 
institution's performance. 

The limitations of many of the commonly discussed performance indicators can 
be demonstrated when applied to a distance teaching university with a modular 
degree structure that does not require students to possess entrance qualifications. 
For example, a large proportion of O.U. students remain in full-time 
employment; many are 'underqualified* for higher education and a considerable 
number achieve the personal or vocational enhancement they desire (change of 
job or role within an organisation, acquisition of new skills, improved 
effectiveness at work, etc.) after completing several courses - but without 
graduating. Adult learners tend to have goals that are not wholly instrumental, 
although currently it seems that extrinsic outcomes of higher education are all 
that are considered worthwhile by central planners (Cullen 1987). 

This section describes briefly the range of 'quality control' procedures that have 
been utilised in the Open University. Most are undertaken on a routine basis 
during the development or presentation of courses; others are used only in 
certain circumstances. Table 1, drawing upon papers prepared for the D.E.S. 
appointed Visiting Committee of the Open University (Kirkwood 1987 and 
Crooks 1988), classifies the routine monitoring procedures according to when in 
the course development process the evaluation takes place and who is 
responsible for making the appraisal. The focus of these procedures is the 
individual courses within a modular structure. 



Table 1: Monitoring and Evaluation Procedures at the Open University, U.K. 



Group making appraisal of Procedures employed 
teaching effectiveness. during course 

development - i.e. before 
presentation to students. 
(Formative Evaluations) 



Procedures employed 
appraisal of during 
presentation of courses 
to students. 

(Summative Evaluations) 



Staff 



Critical commenting by 
members of course team. 

Panel of experienced tutors 
comment on proposals for 
content and presentation. 

Analysis of characteristics 
of students on course being 
replaced or taking related 
course(s). 

Approval of assessment 
materials by Examination 
Board for course. 



Analysis of student 
progress data. 

Tutor de-briefing at end of 
year. 

Routine analysis of grades 
for continuous 
assessments. 

Monitoring of tutor- 
marked assignments by 
course team. 

Standardisation of exam 
marking. 

Conflation of continuous 
assessment and exam 
results by Exam Board for 
course. 



Externa! Academics 



Course Assessors) 
comment on draft materials 
in terms of comer.* «md 
quality of teaching - Repent 
to course team and to Vice- 
Chancellor. 

Approval of assessment 
policy and materials. 



External examiner 
approves assessment 
arrangements and 
materials. 



External examiner(s) 
attend Exam Board 
meetings for all courses 
and submit report to 
university. 

Postal surveys of students' 
reactions to the course. 



Students 



Consumer research of 
potential students. 

[Detailed evaluation of all 
or part of existing course - 
especially if intended for re- 
use.] 

Developmental testing of 
all or parts of course 
materials. 



Interviews with studen s 
(face-to-face or by 
telephone) about teaching 
effectiveness. 

Direct contact between 
course team members 
and students at tutorials 
and at residential 
schools. 



A wide variety of evaluative information is used to shape courses during their 
development and to modify and improve individual courses during then- 
presentation to several cohorts of students. In addition, mere are many occasions 
when it is considered necessary to undertake an evaluation study of a particular 
innovation or component used in a number of courses. Such studies are likely 
to be concerned with drawing out generalisations from the use of a particular 
aspect of the teaching, or with establishing the effectiveness of a particular 
strategy or teaching medium. 

However, our evaluation procedures must go beyond providing just an 
assessment of the instructional materials and situations; they must also 
illuminate the learning and development that occurs both within individual 
courses and over the range of courses students combine to form their 
programme of studies. Research been undertaken into the intellectual 
development of students as a result of their studies; for example, qualitative 
changes in the understanding of key concepts and relationships in the social 
sciences. The development of students as learners (and in other ways) over the 
period of their undergraduate studies has be *t the focus of longitudinal studies 
(Taylor & Morgan 1984 & 1986). 

In Conclusion. 

The demands for greater efficiency in higher education must not be allowed to 
detract from efforts to improve educational quality and effectiveness. 
Maintaining and improving academic standards should involve due 
consideration of the purposes of higher education and concern that the quality of 
student learning is not allowed to remain a forgotten dimension. 
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